What is the impact of words and sentiments in lyrics on People’s attitude towards the songs?

Overview

Music is an important component in the entertainment. Some people say that music may tell us what the artist wants to say, but the popular music tells us what the people want to hear. This project aims to use the sentimental analysis on lyrics of popular music and Twitter posts related to the songs to reveal the secret of the successful songs/lyrics and the impact of the emotions in songs to the audience.

Subquestions:

1. What is the most frequent word used in the Populor Songs?

2. What Have Changed to Popular Songs?

3. Who is the most popular singers in last ten years in Billboard?

4. Is there a pattern for those successful songs?

5. Is there a relationship between the mood in songs with the Twitter Users’ attitudes?

Data Collection and Data Cleaning

Music Data

This project collected the information for music and Twitter posts from different APIs.

The billboard.py, a Python API for accessing music charts from Billboard.com, is used to collect the titles and artists’ names. With the music information, the PyLyrics, a python module to get lyrics of songs from lyrics.wikia.com, helped to find those lyrics.

The biggest problem for the data cleaning in this part is those special signs in the titles or the singer lists:

Graph 1(Tableau)

Besides, there are also some languages other than English in the dataset:

Graph 2(matplotlib), Source Code Graph 3(matplotlib), Source Code

We can see that there are very few songs using languages other than English. The songs with different languages are deleted so that the sentimental analysis will be more accurate.

The comparision between data before and after data cleaning

Graph 4(Tableau)

Twitter Data

To collect Twitter data of singer and song for the past 10 years, we used a Python package called GetOldTweets which enable us to put keywords and date range to collect related tweets. 300 Tweets were collected for each singer and song combination during the week when they appeared on Billboard top 100 songs. Collected Tweets were messy and contain lots of non-alphabetic symbols. And below is the top 20 non-alphabetic symbols in the Tweets.

Graph 5(matplotlib), Source Code

The Hyphen was most frequently used because people are more likely to refer a singer and his/her song together. @ and # are frequently used because they have their actual functionality in Twitter. We could also see that ! has a lot of higher frequency than ?. From my interpretation, it is due to the mood when user sending those Tweets. After listening to the song or hearing related news from some artists, Twitter users are less likely to raise questions, rather Twitter users are more likely to send Tweets to directly convey their emotions (like or dislike) and from sentiment analysis, we could see Tweets with ! has sentiment score average at 0.6.

Twitter data cleaning

When cleaning Twitter data, we first check whether same Twitter post link appear more than once in our dataset, if it does we remove all those duplicate records. Then we removed those non-alphabetic symbols and also hyperlink from text data. And the plot below provides statistics on those records we cleaned.

Graph 6(matplotlib), Source Code

Twitter sentiment analysis

To analyze the sentiment in tweet, we generate a new feature, sentiment score, in the dataset. We build our sentiment analysis tool using Word2Vec and Deep Learning tools. We first obtain our training dataset which has 1.4 million Twitter data with labeled sentiment score from Sentiment140. Then we tokenize both training data and our own Twitter data using NLTK tokenizer. Then vectorize each token using Gensim Word2Vec model with vector size 512 and window size 10. Then construct our 7 layers Convolutional Neural Network model and trained it using our training data.

With 10 fold cross-validation, we gained 75% accuracy and we apply our model to predict sentiment score on collected Twitter data. Which yield frequency distribution as below:

Graph 7(matplotlib), Source Code

1. What are the most frequent words used in Populor Songs?

Let’s start to find our answer by showing some EDA plots. In the first sub-question, this project aims to simply study what is the word used most frequently in the lyrics. We first use the word cloud plot for all the songs in Billboard Top 100 to study it.

Graph 8(matplotlib, Word Cloud), Source Code

According can see that KNOW and LIKE are the most frequent words in the songs. This suggests that there are a lot of songs related to love in the songs. However, we can also see offensive and slang Words like NIGGA and BITCH in the plot. Most of these offensive words indicate hateful thoughts and moods. We furtherly study the lyrics by plotting the world cloud for those songs which had reached to Top 5 in the Billboard.

Graph 9(Matplotlib, Word Cloud), Source Code

The plot shows that there are still a lot of words related to love and romantics. Besides, there are less dirty words in the higher ranking songs, the reason might be that these songs are accepted by more people and people are less acceptable with lyrics full of offensive words.

2. What Have Changed to Popular Songs?

Now, our team wants to study the trend for the development of popular music throughout the years. The plots will show how the names of songs changed, what kind of songs will stay in the Billboard longer:

Source: BillBoard Top 100 Weekly List, 1990 - 2010
Data Grabbed: Song Name, Artist, Number of Weeks Stayed on Billboard, Year, Peak Position, and Lyrics

(1). How did the Song Name Change from 1990 to 2017?

- Calculated the Average Number of Words in Song Names and Visualized by Year

- The Avergae Number of Words Shortened Significantly

- Shorter Song names makes songs more popular

Graph 10(Plotly), Source Code

(2). Is There Any Difference Between the Top 50 Songs and the Bottom 50 Songs?

- This time we calculated average number of words in song names again, but split by Top 50 and Bottom 50

- The Bottom 50, the relatively less popular songs, have more variations

- The Top 50, the most popular songs, have more steady decrease

- For the relatively more popular songs, decrease in length of song names is a trend

Graph 11(Plotly), Source Code

(3). What Songs Are More Likely to Stay Popular for Longer?

- Measured by Number of Weeks Stayed on Billboard

- By Year and Number of Words in Song Names

- In the 90’s, A song with short names is more likely to stay on the list for longer

- The Average number of weeks popular songs can stay on billboard have dropped significantly present todays.

Graph 12(Plotly), Source Code

(4). What Lyrics Made Songs Popular from 2008 - 2018?

- Ranked artists by number of songs from them that were ever on BillBoard

- Ranked the Top 20 most common words used by those popular singers

- The word ‘like’ is the mostly used by the artists

Graph 13(Matplotlib), Source Code

3. Who is the most popular singers in last ten years in Billboard?

First, let us have a look at those artists with the most songs on Billboard’s Top 100 Hits in the last ten years.

Graph 15(Matplotlib), Source Code

Does those singers with more songs count have higher rank in the board?

Graph 16(Tableau)

According to the plot above, the answer is no. Not let us consider what is the relationship between the number of singers’ songs with the number of weeks on the Billboard Top100.

Graph 17(Tableau)

It seems there is a logarithm relationship between the count and the mean total weeks of songs on the Billboard. And the plots suggest that quantatiy in popular music field is negatively correlated to the quality.

4.What is the characters for those popular songs’ lyrics?

In this sub-question, we want to study what kind of lyrics or what sentiment/mood in lyrics is more likely to make the song popular. In the first part of these project, the team studies a sample singer, Talyor Swift, thoroughly and uses EDA plots to study the pattern of the whole dataset which contains the lyrics for all popular songs in the Billboard Top100. These part will lay a foundation for the study in the second part of the project.

(1).Talyor Swift Example

Last sub-question shows that Talyor Swift is one of the most successful singers in the past ten years. Besides, her songs include many different moods or themes. Talyor Swift is a good sample to start our study. We use the lyrics in all Talyor’s songs in this part.

Graph 18(ggplot), Source Code

The average word count for the tracks stands close to 375, and the chart shows that the maximum number of songs fall in between 345 to 400 words. The density plot shows that the distribution is close to a normal distribution.

Graph 19(ggplot), Source Code

The Album-wise word count shows that there tend to be more words in the lastest album.

Graph 20(ggplot), Source Code

This plot shows a similar result. This means, to be a superstar, you need to convey more information in your songs, with which you can influence your audience.

Now let us start the study on the sentiment in songs. The lyrics are first mapped to sentiment score in R and the scores are classified to 8 kinds of moods: Anger, Anticipation, Disgust, Fear, Joy, Sadness, Surprise, and Trust. At the same time, the scores are binned into two categories, representing the positive sentiments and the negative sentiments respectively.

Graph 21(ggplot), Source Code

In the Talyor Swift example, the most frequent mood in the songs is positive. And we can see that Talyor Swift has expressed all kinds emotions in her songs. Joy, anticipation, and trust emerge as the top 3.

Graph 22(ggplot), Source Code

Now that we have figured out the overall sentiment scores, we should find out the top words that contribute to various emotions and positive/negative sentiment. The visualization is given shows that while the word bad is predominant in emotions such as anger, disgust, sadness, and fear, Surprise and trust are driven by the word good.

Graph 23(ggplot), Source Code

We can see that joy has the maximum share for the years 2010 and 2014. Overall, surprise, disgust, and anger are the emotions with the least score; however, in comparison to other years, 2017 has the maximum contribution for disgust. Coming to anticipation, 2010 and 2012 have a higher contribution in comparison to other years.

(2). EDA for the whole lyrics dataset

In the second part of this project, we will furtherly study the trend and the sentiment for all the songs in Billboard just like the Talyor Swift example showed above. In the part of the project, we use some other visualizations graphed by seaborn to see the popularity of the songs and how it changed during the years or even months. These EDA plots could help us to find some interesting features in the dataset.

Graph 24(Seaborn), Source Code

The graph above shows the density of relatively time x-axis and the lengths of weeks that songs stayed on board. It’s easy to tell that the graph has two part of high density, which means in the middle time of these years. Around the year 2014, there are lot great songs been created. Also, around the year 2012, a lot of songs been created but did not stay on the board as long as the songs been created in 2014.

Graph 25(Seaborn), Source Code

The graph above shows the number of favorites and retweets to the songs in ten years. As we can directly see, there is a huge difference between before the year 2017 and after the year 2017. After the year 2017, the number of favorites and retweets increased incredibly. That tells either the songs on the board have more influence after the year 2017 or people started to be crazy about using tweets.

Graph 26(Seaborn), Source Code

The graph above shows the distribution of songs’ popularity each year. As we can see, 2017 has the highest mean, which means most songs been created in 2017 are more popular than other years. However, for 2008, although the high point is really high this year, the mean is low. which means it has both good songs and bad songs this year.

Graph 27(Seaborn), Source Code

The graph above shows the popularity of the songs in each month these years. From the graph, we also expected to see some trends through the changing of the color in the graph. And as the result we can tell from the graph, it did have a slight pattern of the increasing popularity from 2008 to around 2013. However, after that, the popularity of the songs each month became unpredictable.

5. Is there a relationship between the mood in songs with the Twitter Users’ attitudes?

The fifth sub-question is whether there is a relationship between the sentiment analysis results from tweets and lyrics. In other words, the question becomes “Do the lyrics’ attitudes influence people’s comments about them on tweets?” and “whether the result of two attitudes’ connection influence the popularity of the songs?”

In order to answer the questions above, we have to take a look of the results of sentiment analysis from tweets and lyrics.

Graph 28(Seaborn), Source Code

The graph above shows the two-dimensional plot of sentiment analysis attitude scores. As we can see, it seems like there is no linear relationship between the tweets’ attitude and lyrics’ attitude. However, we can not be sure yet until we run some tests. Still, the graph gives us one interesting information. Most points are located on the top and bottom of the graph, which means most of the songs are either too negative or too positive. And those songs have more attention to tweets than the normal songs.

Next, let us take a look at each individual analysis result.

Graph 29(Seaborn), Source Code

As the boxplot shows above, we can see that lyrics have much higher attitude points than tweets. Most of the tweets’ attitude points are below normal, which means they are more negative while most lyrics of the songs tend to be positive. However, we still cannot be 100% sure that they are significant differences, so we will run one-way ANOVA test to testify our thoughts.

Hypothesis test of One-Way ANOVA test.

Hypothesis1, Source Code

As the p-value show above, since the p-value is below 0.05, which rejects the null hypothesis. Then it means there is a significant difference between lyrics’ attitude points and tweets’ attitude points. Since we know they are different, we are moving to the next part, to test whether there is a relationship between the lyrics’ attitude and tweets’ attitude.

Hypothesis test of Linear regression test.

Hypothesis2, Source Code

As the results show above, we used the number of weeks that songs stayed on board as addition term that may relate to the attitude points. Then we found out that all p-values are much greater than 0.05, which means it failed to reject the null hypothesis, and it means there is no significant linear relationship between Tweets’ attitude points and neither lyrics’ attitude points nor the number of weeks that songs stayed on board( which may also be thought as popularity).

Through the hypothesis tests we did above, we found out that tweets’ attitude and lyrics’ attitude of the songs have significant differences, and there is no significant linear relationship between them. However, that does not tell everything between them. Adding other possible related terms and use other forward tests may still get some good results. So, in the next part of the project, we gonna dig much deeper to discover the possibilities that they connect with each other, and how they may influence each other in any possible ways.

Conclusion and Future Study

In the first part of this project, the team collects the data for Music Lyrics and Twitter and finishes the sentimental analysis on the lyrics and tweets. The EDA plots are used to partially answers the five sub-questions. In the second part of the project, our team will further study these successful popular songs and the words/sentiments in the lyrics. We will also try other methods to find out the patterns for the audiences’ attitudes on the songs.